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Summary 


Asynchronous  training  technologies  enable  students  to  master  materials  much  more  rapidly 
than  classroom  instruction.  Students’  time  saving  represents  a  major  potential  benefit  of  using 
them. 


This  paper  fills  a  methodology  gap  in  estimating  the  students’  timesavings  benefit  of 
asynchronous  training  technologies.  Meta-analyses  of  their  effectiveness  yield  a  statistic  called 
Effect  Size.  Estimating  the  benefit  in  dollar  terms  requires  information  regarding  the  reduction 
in  total  training  time.  This  paper  presents  a  methodology  for  interring  the  percentage  reduction  in 
students’  training  time  from  estimates  of  Effect  Size.  It  goes  on  to  infer  the  percentage  reductions 
for  three  asynchronous  training  technologies  from  estimates  of  their  Effect  Sizes.  Finally,  it 
compares  the  estimates  to  some  direct  measures  of  training  timesavings. 


Inferring  Saving  in  Training  Time 
from  Effect  Size  Estimates 


I.  Introduction 


Since  at  least  the  1970’s,  cognitive  psychologists,  computer  scientists,  and  educators  have 
been  developing  software  technologies  for  training.  Moreover,  they  have  shown  that  many  of  these 
technologies  are  significantly  more  efficient  that  standard  classroom  instruction. 

Users  can  reap  the  benefits  of  more  efficient  instructional  technologies  either  as  increased 
student  achievement  or  as  reduced  training  time.  Most  empirical  evaluations  have  expressed  the 
efficiency  gains  in  terms  of  achievements.  This  technical  paper  lays  out  a  method  for  using  this 
data  on  achievement  gains  to  estimate  reductions  in  training  times.  For  each  instructional 
technology,  the  method  can  provide  a  curve  for  trading-off  achievement  and  training  timesavings. 

Several  researchers  have  provided  reviews  and  analyses  of  the  empirical  studies 
documenting  the  achievement  gains  and  training  timesavings.  They  include  Orlansky  and  String 
(1979),  Kulik,  Kulik,  and  Shwalb  (1986),  Niemiec  and  Walberg  (1987),  Fletcher  (1990),  Redding 
and  Fletcher  (1994),  Metzko,  Redding,  and  Fletcher  (1996),  and  Regian,  Seidel,  Schuler,  and 
Radtke  (1996).  These  reviews  and  analyses  show  remarkably  consistent  results  for  particular 
instructional  technologies.  This  paper  uses  their  results  and  my  method  to  estimate  training 
timesavings  for  three  asynchronous  training  delivery  technologies. 
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n.  Methodology 


Effect  Size.  The  analyses  and  reviews  have  made  empirical  studies  comparable  by 
converting  them  to  an  effect  size  (E)  metric.  For  our  purposes,  it  can  be  defined  as  the 
standardized  mean  difference  in  achievement  of  the  group  trained  with  asynchronous  instructional 
technology  (Af)  and  the  group  trained  with  standard  classroom  instruction  (A^).  Thus, 

Et  =(At -Ac)/Sc  (1) 

where  is  the  standard  deviation  of  the  group  trained  with  stand  classroom  instruction. 

Time  Savings.  Holding  instructional  time  constant,  the  relative  achievement  of  students 
training  with  technology  t  is  At  /Ac ,  which  is  assumed  to  be  equal  to  or  greater  than  one.  So, 
students  would  learn  in  l/(  At  /Ac  )  hours  what  they  had  learned  in  one  hour  of  classroom 
instruction.  Thus,  the  proportional  time  savings  ((Tc  -  Tt )/  Tc  )  is 

!-(!/(  At /Ac)) 


or 


l-(Ac/At) 


or 

(At  -  Ac  )/  At  (2) 

Using  Expression  1  to  eliminate  the  expression  At  -  Ac  from  Expression  2,  we  get  the  following  for 
the  proportional  time  savings  due  to  using  instructional  technology  t, 

(Et  Sc)/ At  (3) 

We  can  Expression  3  to  estimate  time  saving  for  instructional  technology  t. 

The  estimates  of  effect  size  are  based  on  normalized  standard  deviation  of  0.34  (Sc  =  0.34). 
Substituting, 


(Tc  -Tt)/Tc=  0.34  Et/ At  (4) 

Algebraic  manipulations  gives  us 

Tt  =Tc  -  (034EtTc)/At  (5) 
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If  we  focus  on  an  hour  of  classroom  instruction,  we  can  write  Tg  -  60  minutes.  Substituting, 

Tt  =  60  -  (20.4Et/At)  (6) 

Equation  6  describes  a  trade-off  relationship  between  training  time  for  technology  t  (Tt )  and 
desired  average  achievement  using  technology  t  (At).  The  only  parameter  is  the  effect  size  of 
technology 
t(Et). 


The  estimates  of  effect  size  are  based  on  a  standardized  mean  achievement  for  the 
classroom  group  of  0.5  (Ac  =  0.5).  An  assumption  of  unchanged  achievement  implies  that  At  =  0.5. 
Substituting  in  Equation  (4),  we  get  the  following  equation  for  proportion  timesavings: 

(Tc -Tt)/Tc=  0.68Et  (7) 

Substituting  in  Equation  (6),  we  get  the  following  equation  for  the  number  of  minutes  of 
instructional  time  required  to  provide  the  same  amounts  of  training  and  achievement  as  one  hour 
of  class  room  instruction: 


Tt  =  60  -  40.8  Et 


(8) 
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ni.  Estimates  of  Time  Savings 


Interactive  Courseware  (ICW)  provides  instructional  materials  to  students  individually  (1). 
Instruction  is  self-paced.  ICW  provides  interactions  that  tailor  the  instruction  to  the  needs  of 
individual  students.  Each  student  receives  that  level  of  detail,  pace,  remediation,  sequence  of 
topics,  and  interactions  needed  to  learn  the  material.  ICW  encompasses  computer-based 
instruction  (CBI),  CD-ROM  instruction,  and  interactive  videodisc  instruction. 

Researchers  have  made  much  progress  in  estimating  the  effect  size  of  ICW.  Redding  and 
Fletcher  (1994,  pp.  75-6)  discuss  meta-analyses  and  reviews  of  CBI  effectiveness.  They  find  that 
the  most  relevant  meta-analyses  and  the  broadest  reviews  suggest  an  average  effect  size  of  0.42  for 
CBI.  Fletcher  (July  1990,  p.  III-IO)  gets  an  average  effect  size  of  0.44  in  his  meta-analysis  of 
interactive  videodisc  instruction.  In  a  forthcoming  meta-analysis  of  ICW  in  military  training,  the 
Institute  for  Defense  Analysis  (IDA)  will  report  an  effect  size  of  0.44  (Metzko,  Redding,  and 
Fletcher,  December  1996,  p.  D-3). 

Through  based  on  different  data  sets,  these  estimates  of  effect  size  are  remarkable  close. 
Let’s  use  an  effect  size  of  0.44  to  compute  our  estimates  of  timesavings.  Substituting  this  value  into 
Equations  7  and  8,  we  get  an  average  timesaving  of  30  percent  for  ICW  and  an  average  time  of  42 
minutes  to  provide  the  equivalent  of  an  hour  of  conventional  instruction.  This  average  time  savings 
is  consistent  with  Orlansky  and  String’s  (April  1979,  p.  5)  finding  that  students  instructed  by  CAI 
save  about  30  percent  of  the  time  required  to  complete  the  same  courses  given  by  conventional 
instruction. 

The  timesavings  provided  by  ICW  may  arise  primarily  from  its  self-paced  character,  which 
frees  more  able  students  from  the  common  pace  of  the  classroom.  So,  ICW’s  timesavings  may 
reflect  primarily  the  more  able  students  getting  through  the  materials  more  rapidly. 

Tutorial  ICW.  In  his  1990  study,  Fletcher  (pp.  m-ll  to  m-15)  found  that  ICW  that  used  a 
directive  and  tutorial  approach  has  significantly  large  effect  sizes.  (The  reader  should  not  confuse 
Tutorial  ICW  with  intelligent  tutors.  None  of  this  instructional  courseware  had  student,  expert,  or 
instructional  models.  The  tutor  components  of  many  standard  software  programs  are  examples  of 
tutorial  ICW.)  He  estimated  an  average  effect  size  of  0,68  for  tutorial  ICW.  Substituting  this  value 
into  Equations  7  and  8,  we  get  an  average  timesaving  of  ^  percent  for  Tutorial  ICW  and  an 
average  time  of  32  minutes  to  provide  the  equivalent  of  an  hour  of  conventional  instruction. 

Intelligent  Computer  Aided  Instruction  (ICAI).  Evidence  on  the  effect  size  of  ICAI  is  much 
more  limited.  Only  three  studies  have  provided  estimates  (Regian,  Seidel,  Schuler,  and  Radtike, 
1996,  pp.  11-13).  All  three  are  very  close  to  one.  Substituting  this  value  into  Equations  7  and  8,  we 
get  average  timesavings  of  ^  percent  for  ICAI  and  an  average  time  of  19  minutes  to  provide  the 
equivalent  of  an  hour  of  conventional  instruction.  This  estimate  of  average  timesavings  may  be 
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high.  Regian,  Seidel,  Schuler,  and  Radtike  report  average  timesavings  of  55  percent  for  three 
evaluations  of  ICAI  in  higher  education. 
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IV.  Using  the  Estimates  in  Benefit/Cost  Analysis 


Reductions  in  training  time  have  implications  for  estimating  both  the  benefits  and  the  costs 
of  using  asynchronous  instructional  technologies.  On  the  benefit  side,  they  mean  that  less  student 
and  instructor  time  must  be  devoted  to  training.  (Office  of  the  Under  Secretary  of  Defense  (P&R)) 
This  savings  may  represent  the  quantitatively  most  important  benefit  of  using  the  new  technologies. 

'On  the  cost  side,  development  hours  per  hour  of  instruction  drive  the  cost  of  developing 
courseware  and  materials.  The  more  efficient  instructional  technologies  require  more  development 
hours  per  hours  of  instruction.  However,  their  greater  efficiency  means  that  fewer  hours  of 
instruction  are  required  to  provide  a  given  amount  of  training.  This  reduction  will  offset,  in  whole 
or  in  part,  the  greater  number  of  development  hours  per  instructional  hour. 
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